We use helm to add the relevant chart from nvidia and install the chart.
We re-use existing script to verify the rollout too.

This job needs to run after either one of the two jobs above it enabling
NVIDIA GPU in the k8s cluster succeed (one is for microk8s, the other is
for Canonical k8s).  We can't use those jobs in 'depends' because then
both of them will have to succeed, which is impossible because only one
of either microk8s or Canonical k8s will be available.

The trick we use here is that we now `depends` on `dss/initialize`,
which must succeed for the whole test-plan to be run anyway, and,
we require that we have an NVIDIA GPU.  This is similar to the `depends`
for the two jobs for microk8s and Canonical k8s.  Then we will have to
be careful that this job is added in the test-plan to ONLY run after
those two jobs.  The difference will then be that this job will not be
skipped if either of the two jobs enabling NVIDIA GPU fail.

We now have an addition to the `install-deps` script. It also demarcates
from whence we started supporting Canonical K8s

The "worker" daemonset that was being verified may have a version number
in its name, which we cannot predict.

We do not need to customize the slots_per_gpu today, so let's keep it
very simple, and directly apply the Kustomize configurations

KUBECONFIG is actually used by kubectl, and other tools, and this is not
the right use of it

Modified from copy of checkbox-ce-oem native build

Currently only requires passing environment variables pointing to the
containerd config and socket paths for microk8s.  This customisation is
not required on Canonical K8s.

The original job is renamed and should continue to work.  We need a
slightly different job for installing the gpu operator on microk8s.

This reverts commit 02138ef.

We are going to detect it

The labels take some time to propagate

If there's a TimeoutError, `microk8s status` was still executing, so it
is there ... even thought it does not tell us whether microk8s is in use
or not.  Anyway, throw the error, instead of deciding that there is no
microk8s.

The validator container may not have been created even after the
daemonset is rolled out (for some unknown reason), hence we wait before
checking the logs.  And, since checking the logs will wait for the
validations so succeed, this job may take considerably longer.

It is FileNotFoundError that is raised when microk8s is not installed.
So we are not going to try catch all other CalledProcessErrors, for now.

we need snapd > 2.59 for SNAP_UID